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ABSTRACT 

A study explored the reliability of methods for 
analyzing the thematic organization of texts. Five linguists, two of 
whom were native speakers of Swedish and three of whom were native 
speakers of Finnish, analyzed the thematic organization of a Swedish 
popular scientific article, using an analysis that focused on 
coherence reflected in the thematic progression of the text. The 
analysis was derived from the experimenter's model of thematic 
relations. The five subjects' interpretations concerning relations 
expressed in the utterances were compared for extent of agreement. 
Results suggest that, while the sample of subjects and the quantity 
of text used were very limited, the categories of textual relations 
used for analysis are sufficiently clear to make the kinds of 
judgments required here. Contains 19 references. (MSE) 
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SOME CONSIDERATIONS OF INTERSUBJECTIVITY IN 
THE ANALYSIS OF TEXTUAL RELATIONS 



Merja Koskela 



1. Introduction 

Thematic organisation and textual coherence are perhaps the two most 
frequently used approaches to text analysis. These approaches are closely 
interrelated and they both depend on a subjective, knowledge-based 
interpretation on the part of the reader during the reading. It has 
therefore proved difficult to find objective and intersubjectively 
verifiable models for this kind of analysis. Most analyses of these aspects 
are necessarily intuitive and, at best, examples of the text are cited to 
justify the researcher's findings. 

The interrelationship between thematic organisation and coherence is 
based on the idea that the utterances which constitute a text must 
necessarily have some connection with each other, explicitly on the 
surface level (cohesion) and/or more implicitly on the conceptual level 
(coherence) 1 , if the text is to fulfil its particular communicative goal. 
In other words, for us to understand a text, it has to be coherent on some 
level, otherwise it will not be communicatively adequate. In this way, the 
need to create coherence determines many structural and functional 
choices in communicative language use. A result of one of these choices 
is the thematic organisation of texts. 

In a study where I compare Swedish scientific and popular scientific 
texts, I use a method which combines thematic organisation with an 
analysis of semantic and pragmatic textual relations. The relations 
analysed are partly surface phenomena and partly conceptual ones. Both 
types of textual relations are equally important for a reader trying to 
interpret the thematic organisation of the text. 

The method is originally based on the types of thematic progression 

* Cohesion in a text is created by explicit textual relations on text 
surface, whereas coherence includes the more implicit textual relations 
as related to the background knowledge, beliefs and assumptions of the 
reader (cf. de Beaugrande & Dressier, 1983; Lundquist, 1989: 123). 
However, even the surface form of a text can be seen as contributing to 
coherence in the text (see e.g. Hasan, 1984). 
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presented by Danes (1974), but it is supplemented with a systematic 
categorisation of the textual relations that lie behind each type of 
thematic progression (for a similar approach see Scinto, 1983). As such, 
the method is just as intuitive as any other method concerning the 
thematic organisation of texts, and it is therefore useful to consider how 
subjective or intersubjective the results gained with this kind of a method 
can be. 

Intercoder reliability is often discussed in linguistic research 
(Tirkkonen-Condit & Lieflander-Koistinen, 1989: 176; Mauranen, 1993: 
49; Lauren, 1993: 158f), but it is seldom explicitly tested. However, the 
lack of a strict objective method of analysis should invite researchers to 
explore just how subjective their results are bound to be. For this 
purpose I have conducted a pilot study, where on the basis of a relatively 
restricted categorisation of textual relations, four test subjects besides 
myself analysed intersentential relations relevant to the thematic structure 
of texts. The aim of the study was twofold: fir&t, to evaluate how 
intersubjective the results of this kind of analysis are, and second, to find 
out whether the categories used are clear enough to function as a basis of 
analysis. 

2. The method of analysis 

The method of analysis used in my study is based on the idea that the 
thematic organisation of texts is governed by the notion of coherence. 
However, as far as I know, these approaches have not been systematically 
related to each other before. The only explicit attempt in this direction 
which I have come across is that of Scinto (1983: 84f; see also Lundquist, 
1989), even though there has been a repeated demand for such an 
approach (see e.g. Giilich & Raible, 1977; Hoffmann, 1989: 222). Other 
attempts in this direction have been made by e.g. Gerzymisch-Arbogast 
(1987) and Makovec-Cerne (1992), but with little discussion of the 
explanatory power of such an approach. 

Most of the work done on relating thematic organisation to the notion 
of coherence is based on Danes' model of thematic organisation of texts 
(1974). He proposes that the thematic organisation of texts be described 
by means of types of thematic progression. The method was originally 
concerned with sentences, but was later also successfully applied to 
longer stretches of text, indeed because the types of thematic progression 
are ultimately based on coherence. Danes (1974: 114) defines the concept 
of thematic progression as 
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"...the choice and ordering of utterance themes, their mutual 
concatenation and hierarchy, as well as their relationship to the 
hyperthemes of the superior text units (such as the paragraph, 
chapter,...), to the whole text and to the situation." 

This definition does not, at least in my opinion, limit the principle of 
thematic organisation to single utterances, or to the relationships between 
successive clauses, although many analysts, often for reasons of 
convenience, have wanted to interpret it in this way. 

According to the above definition, the types of thematic progression 
are based on the idea of textual relations in much the same way as 
cohesion, and partly even coherence, are often described. 

However, all processes creating cohesion and coherence in texts are 
not relevant to the types of thematic progression. Rather, the idea is 
based on the following relations: 

1. relations between the abstract notion of textual theme and its 
realisations, the concrete thematic elements; 

2. relations between thematic elements, which occur in successive 
or fairly close sentences, and relations between theme-rheme- 
nexuses and themes; 

3. relations between rhemes and themes (when a rheme recurs 
as the theme of a sentence later on in the text). 

The overall coherence of a text is, however, the result of a complex 
network of different factors that work towards the same goal: for the 
text to fulfil its communicative purpose. The processes that underlie the 
thematic organisation and thus create coherence make it, for their part, 
possible for the reader to experience the text as a connected whole. As 
mentioned before, these processes include both cohesion level and 
coherence level phenomena. However, the line between cohesion and 
coherence is not always easy to draw. 

Looking at thematic progression at sentence level gives a picture of 
the local linear structuring of the text. This kind of analysis can be used 
to account for some characteristic features of different kinds of texts, 
e.g. texts representing different genres, but it does not describe the 
global thematic structure of whole texts. 

If we want to get a more comprehensive idea of the thematic structure 
of whole texts we have to analyse larger entities than single sentences or 
utterances. Both the cohesion and the coherence levels of description of 
thematic organisation have to be accounted for. 

In order to achieve this I have created a preliminary categorisation of 
textual relations and their connections with types of thematic 
progression, in a manner similar to that proposed by Scinto (1983). 
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The categorisation is based on the work done by Gerzymisch- 
Arbogast (1987) and Makovec-Cerne (1992), as well as on my own 
experiments based on Swedish scientific and popular scientific texts. It 
should be noted, however, that the categories I propose are not 
necessarily exhaustive or even mutually exclusive. Such a claim cannot be 
made until a fully adequate theory of textual relations for intrasentential 
linkage exists. 

Figure 1. Textual relations categorised according to thematic patterning. 
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TP1 (simple linear thematic progression) and TP2 (thematic 
progression with a continuous theme) represent the traditional sentence- 
level approach, the cohesion-level of analysis, adopted by many scholars 
(see e.g. Wiegand, 1988; Francis, 1989; Dubois, 1987; Mauranen, 1993). 
The textual relations that lie behind these types of progression are direct 
semantic relations. The mosv usual ones are listed in the figure (identity, 
synonymy, contrast, pronominalisation). 

It should also be noted that these relations need not connect adjacent 
sentences in order to be relevant. A span of 4-5 sentences or even longer 
is possible. Givon (1983: 13) for example considers 20 clauses the limit 
of referential distance. 

TP3 represents my understanding of what is commonly known as 
hypertheme-progression. This type is based on indirect semantic relations 



(the coherence level), for example hyponymylhyperonymy, 
part/whole, co-hyponymy and antonymy, or on semantic inference. 

TP 4 is my addition to Danes' types and it accounts for the themes that 
lie outside the immediate scope of the text. This kind of thematic 
development, here termed semantic-pragmatic relations, includes e.g. 
references to time or place, or to the writer, as well as some grammatical 
constructions that are often thematised (for example the Swedish pronoun 
man) {inference of a semantic-pragmatic type). 

As the above description of the method shows, there is no doubt room 
for intuitive judgement in the analysis. Therefore, the suitability of the 
categories used and the intersubjectivity of the results have been tested in 
the pilot study I will describe below. 

The method even makes it possible to relegate the thematic elements 
to different categories of thematic progression on the basis of the textual 
relations. However, in this pilot study I will only concentrate on 
discussing the intersubjectivity of the judgements concerning textual 
relations because the material used is not large enough for an analysis of 
types of thematic progression. 

Usually tests of intersubjective agreement reflect intraobserver 
inconsistencies and interobserver differences, but even differences in 
interpreting the instructions may have considerable impact on the results. 
Hence, I do not claim that an intersubjectivity comparison of a small 
number of testees necessarily yields valid results. As Mauranen (1993: 
49) puts it: "individual bias is not radically different from the bias of two 
or three individuals in text interpretation." 

However, the test is of interest as a linguistic experiment because 
linguistic methods are far too seldom tested in this way. Additionally, a 
careful interpretation of the results of the test should give at least some 
guidance as to how suitable the categories established to describe the 
textual relations relevant to thematic organisation are, and how reliable 
the results of an analysis of this type can be. 



3. The choice of test subjects 

The test subjects were four linguists, all female, as well as myself. Two 
of the linguists are native speakers of Swedish, and two besides me have 
completed university studies in Swedish (mother tongue Finnish). The 
main reason for using expert subjects for the test was that teaching 
linguistic categories to "naive subjects" would have taken too much time 
and included too many uncontrollable factors. Using naive subjects would 
require some other kind of experiment design, for example a 



a 

ERIC 



questionnaire, but it is often difficult to say whether such tests really 
measure what they set out to measure. 

To make sure that the group tested was not too homogeneous, I chose 
people with different linguistic interests. Their areas of specialisation 
vary from text linguistics to languages for specific purposes, literature 
and teaching second language courses. 

4. Material for the study 

As material for the study an extract from a Swedish popular scientific 
article (Staffan Bergwall: Den svdra konsten att avbilda atomer) was 
used. The particular extract was chosen because it forms an independent 
whole with its own subheading. On the basis of a preliminary analysis the 
extract seemed to be sufficiently representative of different kinds of 
textual relations. And what is more, the extract is of optimal length. This 
kind of analysis must be concerned with an extract that is short enough to 
allow full concentration throughout the text, but long enough to offer 
examples of different kinds of textual relations. In conformity with this, 
the extract chosen for the experiment consisted of utterances 22-45 in the 
article, altogether 24 utterances (see Appendix 2). 

The test subjects were given half an hour to complete the analysis, but 
two of the subjects slightly exceeded the rime limit. This indicates that the 
length of the text extract was suitable. 

5. Experiment design 

The experiment began with instructions given to the test subjects in their 
mother tongue (two times in Swedish, two times in Finnish) individually 
because the tests were carried out at different times during the same 
week. 

First I explained to the subjects what the goals of my research were 
and how the test was related to those goals. Then we discussed the 
categories of textual relations used in the analysis in detail. The testees 
were given one sheet where the categories were presented, and a separate 
sheet with definitions and examples of typical cases (see Appendix 1). 
The instruction part included reading both papers together with the test 
subjects. The subjects were also allowed to use the sheets as an aid during 
the test. A specimen analysis where the test subject's participation 
increased as the analysis proceeded was carried out with each subject 
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before the test. 

At the beginning of the test, the test subjects were presented with the 
whole article from which the test material was an extract. After reading 
through the article, the subjects were given a preanalysed version of the 
text where the thematic elements were marked. The test subjects were 
then asked to follow the instructions, that is, to find and name the textual 
relations relevant to thematic organisation. The test subjects were also 
asked to name the origin of the reference (the thematic or rhematic 
element in question). 

The answers were written on a separate sheet. The interpretation of 
the results is based on these sheets. 



6. Results 

Because of the small number of testees and the small text corpus, 
statistical tests would not yield any useful results. Therefore I will use 
percent agreement to report the consistency of the results even though 
this is a less informative measure than a genuine statistically based 
reliability measure would be (see e.g. Passonneau & Litman, 1993). 
Consequently, the results of this pilot study must be seen as indications, 
not as proven facts. 



Total agreement 


12 


50% 


Almost total agreement (4/5) 


8 


33,3% 


No agreement 


4 


16,7% 


I 


24 


100% 



Table 1. Agreement concerning utterances 

Table 1 describes the consistency of the subjects' estimates of textual 
relations. The table shows that there was total agreement in 50% of the 
24 cases and almost total agreement (when 4 out of 5 test subjects made 
the same decision) in 33,3% of the cases. Together these figures indicate 
that the overall agreement was good: in 83,3% of the cases there was 
either total or almost total agreement between the subjects' judgements 
on textual relations. This figure can be considered high because of the 
general subjective nature of this kind of judgement. 

Kallgren (1979: 80) has stated that 'stronger' textual relations, for 
example identity, are easier to agree on than weaker ones, for example 



inference. Consequently, we could assume that the subjects would most 
often agree on categories identity, synonymy, contrast and 
pronominalisation, and mere seldom on the other categories, except the 
semantic-pragmatic type of inference, which was quite clearly defined 
in the instructions (see Appendix 1). The results of the pilot study 
support these assumptions: 



Type of relation 


Total agreement 


Almost total agreement 


Identity 
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Contrast 
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Pronominalisation 


0 
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Semantic inference 


0 


2 


Sem-pragm. inference 


4 


0 


Total 


12 


8 



Table 2, Total agreement and almost total agreement according to the 
type of textual relation. 

Identity and the second type of inference (semantic -pragmatic) seem 
to be the strongest textual relations among the categories. When we look 
at the 20 cases where there is either total or almost total agreement, 
identity comes out as the category with the highest degree of agreement: 
there is total agreement 6 times out of 12 and almost total agreement 4 
times out of 8. On inference of the semantic-pragmatic type there is 
total agreement in 4 cases out of 12. 

Considerable disagreement among the test subjects was found in 4 
utterances out of a total of 24 (that is 16,7%). A closer look at these cases 
reveals, that there are several textual relations involved. For example for 
the beginning of the text extract (utterance 22), three different categories 
of textual relations were suggested, even though the instructions 
explicitly stated that the beginning of a chapter is usually categorised as 
semantic inference from the textual theme. Consequently, the 
differences here can be counted as misinterpretations of the instructions. 

The other cases of disagreement are more interesting. In spite of the 
fact that the textual relations in question do not seem to be difficult for 
the reader to understand, different categories were proposed. Utterance 
25 (below) serves as an example of how difficult it is to create mutually 
exclusive categories for textual relations (thematic elements printed in 



9 



boldface): 

24. och upphovsmannen Gerd Binning och Heinrich Rohrer fran IBMs 
iaboratorium i Schweiz fick halva Nobelpriset i fysik ar 1986 for sin insats. 

25. (Andra halvan gick till uppfinnaren av elektronmikroskopet.) 

The rheme in utterance 24 states that two scholars got half the Nobel 
prize for their invention in 1986, and the theme in utterance 25 explains 
that the other half of the prize went to the inventor of the electron 
microscope. 

This relation was described as identity, contrast (by two subjects), 
part/whole relationship and as co-hyponymy. Obviously, on some level 
all the subjects are right in their judgements, even though identity in this 
case is perhaps a somewhat unusual interpretation and part/whole 
relationship has probably been chosen because of the semantic content of 
the elements involved in the relation (half, the other half). 

The rest of the disagreements (utterances 32 and 41; see Appendix 2) 
were, as could be expected, between the two types of inference: 
semantic and semantic-pragmatic inference. It is sometimes difficult to 
decide whether a thematic element should be considered something 
closely related to the textual theme, in this case a microscope (semantic 
inference) or as something from outside the scope of the text (semantic- 
pragmatic inference). This kind of confusion was, for example, caused 
by utterance 41 (below), which introduces a computer (en dator) in the 
description of the method of how the microscope functions: 

40. Information om denna variation nar nalspetsen som hela tiden "flyter" pa ett 
och samma av stand ovanfor ytans struktur. 

41. En dator ritar sedan upp en bild av den undersokta ytan (bild 1). 

If we want to get an idea of how individual test subjects agreed with 
each other, we will have to describe the results in relation to majority 
judgements. In studies where no judgements can be considered to be right 
or wrong, this is one way to compare the individual subjects' results. 
However, as the example jusi discussed shows, majority judgements in 
this study should not be taken as true values. 

Table 3 describes the test subjects' agreement with the majority 
opinion, which in this case means that 3 subjects out of 5 agree on a 
given judgement. The percent agreement is the ratio of observed 
agreements with the majority opinion to possible agreements (n=24). 
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Subject Agreement (3/5) % (n=24) 



1 


21 


88% 


2 


18 


75% 


3 


18 


75% 


4 


20 


83% 


5 


21 


88% 


Mean 


19,6 


81,7% 



Table 3. Agreement with the Majority Opinion. 

The table shows that all the subjects agree in at least 18 cases with the 
majority opinion. Subjects 1 and 5 have die highest agreement figures, 
whereas subjects 2 and 3 have the lowest values. Figure 2 below 
illustrates how the judgements made by subjects 2 and 3 relate to those 
made by subject 1. 

Figure 2. The use of different categories by test subjects 1, 2 and 3. 
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A closer look at Figure 2 shows that subjects 2 and 3 differ more in 
their use of the scale than the rest of the subjects do. Subject 1, for 
example, identifies five different categories whereas subject 2 identifies 
six and subject 3 seven. The small corpus, of course, prevents any 
definite evaluations, but it is interesting to note that subjects 2 and 3 do 
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not seem to agree with respect to their use of additional categories. 
Subject 2 prefers horisontal hierarchical relations (categories 7 and 8 in 
the figure), whereas subject 3 prefers vertical hierarchical relations 
(categories 5 and 6). 

Obviously, the causes of differences of judgement can be many. 
Besides differences in knowledge of the world, one possible factor is 
differences in reading strategies. The analysis required by my 
instructions is mainly a surface level analysis where deeper semantic 
implications are not taken into consideration. What Lundquist (1989: 
134) calls coherence-for-an-expert is therefore outside the scope of the 
analysis. In other words, the analysis covers the cohesion level of textual 
relations and that part of the coherence level which does not require 
advanced expert knowledge. The reason for this is the need to gain 
similar results for both scientific and popular scientific writing. Because 
a layman cannot have sufficient knowledge of the different implicit 
lexical relations used in scientific writing, a surface level analysis is the 
only possibility, even though a deeper analysis of semantic relations is 
possible in popular scientific writing (cf. Myers, 1991). However, if the 
test subject's normal reading strategy includes deep semantic analysis of 
the contents of the text and the relations within it, it is likely that this will 
show in the analysis as a wider range of categories. One additional factor 
that supports this interpretation is that subjects 2 and 3, who used more 
categories, are both native speakers of Swedish. This could lead to the 
assumption that a surface level analysis may be easier to conduct for a 
non-native speaker, at least when it is a question of popular scientific 
texts. However, the results only concern two test subjects and are 
therefore not reliable enough for this kind of conclusion. 

The relatively high degree of agreement between the test subjects 
allows us to assume that the method of analysis would guarantee at least 
some degree of reliability even for types of thematic progression. In fact, 
the degree of agreement regarding types of thematic progression should 
be even higher. For example, it does not matter whether a relationship is 
judged as identity or pronominalisation because both are categorised 
either as TP1 or TP2 depending on the origin of the reference. 

7. Conclusion 

In summary, it must be stated that both the number of test subjects 
and the number of cases in the pilot study are too small for any definite 
conclusions. Still, the results of the study indicate that the categories used 



are sufficiently clear to serve as a basis for judgements on textual 
relations, even though there will always be difficulties in making the 
categories mutually exclusive. 

The results concerning the intersubjectivity of the judgements in this 
study reflect interobserver differences. These can be caused by many 
factors, e.g. personal differences, the different subjects' varying 
knowledge of the world or different reading strategies, but they can also 
reflect different interpretations of the instructions. All these are 
uncontrollable factors in any test. Anyway, the result, total or almost 
total agreement in 20 (83,3%) out of 24 cases, seems to encourage 
further investigation with the method. 

However, the intuitive aspect can never be totally removed from the 
analysis of coherence or from the analysis of thematic organisation. Both 
methods call for a qualitative individual analysis where the criteria used 
are made explicit. The agreement or disagreement between two or three 
people does not radically change the picture. 
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